Adaptive quality image reconstruction via a compressed sensing framework

ABSTRACT

What is disclosed is a system and method which reconstructs an N-pixel image of a scene such that Q pixel locations associated with identified regions of interest in a scene have a higher image quality when rendered relative to other pixels in the image. Acquisition and adaptive-quality compression are performed simultaneously by semi-synchronously or asynchronously temporally modulating an ordered set of sampling functions used to spatially modulate a pattern of light. The teachings hereof improve compression efficiency of a compressed sensing framework while improving encoding efficiency with respect to traditional compressed sensing techniques.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation-in-part of commonly owned andco-pending U.S. patent application Ser. No. 13/932,791, entitled:“Reconstructing An Image Of A Scene Captured Using A Compressed SensingDevice”, by Bernal et al.

TECHNICAL FIELD

The present invention is directed to systems and methods which use acompressed sensing framework to reconstruct an image of a scene whereinpixel locations associated with identified regions of interest in thatscene have a higher image quality when rendered relative to other pixelsof the reconstructed image.

BACKGROUND

Compressed sensing is a relatively new area in the signal processing artwhere one measures a small number of non-adaptive linear combinations ofa signal. These measurements are usually much smaller than the number ofsamples that define the signal. From the small numbers of measurements,the signal is reconstructed by a non-linear process which aims to reducethe overall complexity required by a large variety of measurementsystems by introducing signal compression into the measurement process.Essentially, the theory behind compressed sensing is that sparse signalstatistics can be recovered from a small number of measurements. Thesparse nature of most signals of interest allows high-fidelityreconstructions to be made using a compressed sensing approach.Compressed sensing can be beneficial because it reduces the number ofsamples required to spatially and/or temporally reconstruct a givenscene thereby enabling the use of inexpensive sensors with reducedspatial and/or temporal resolution in certain applications where complexsensors are otherwise used, while maintaining the quality of thereconstructed image of the scene. Compressed sensing also holds anadvantage in fidelity over conventional camera systems due to detectornoise issues that may affect measurements due to the limited radiometricefficiency of pixels in two-dimensional sensors. The present applicationprovides a desirable extension to the prior compressed sensing systemtaught by Bernal et al.

BRIEF SUMMARY

What is disclosed is a system and method which reconstructs an N-pixelimage of a scene such that Q pixel locations associated with identifiedregions of interest in a scene have a higher image quality when renderedrelative to other pixels in the image. Acquisition and adaptive-qualitycompression are performed simultaneously by semi-synchronously orasynchronously modulating temporally an ordered set of samplingfunctions, used to spatially modulate light incoming from the scene. Theteachings hereof improve compression efficiency of a compressed sensingframework. Encoding efficiency is improved with respect to traditionalcompressed sensing techniques.

In the context of the following discussion, bold Greek letters refer tosets of functions of a given length and to matrices formed by stackingrows, where each row is a function in the set.

In one embodiment, the present method for reconstructing an N-pixelimage of a scene captured using a compressed sensing device involvesperforming the following. First, a mask is received which identifies atleast one region of interest (ROI) in a scene having Q pixels, whereQ<N. The mask can be a binary image wherein pixels with value ‘1’ (ONpixels) indicate locations associated with the ROI and pixels with value‘0’ (OFF pixels) indicate locations not associated with the ROI.Consequently, the mask has at least Q ON pixels, and at most N−Q OFFpixels. The mask can also be multi-level, each level corresponding toone identified ROI, each identified ROI potentially having a differentquality of reconstruction assigned to it. Next, a set of M samplingfunctions is arranged in an M×N matrix φ={φ₁, . . . , φ_(M)}, whosem^(th) row vector φ_(m)ε

^(N) denotes the m^(th) N-dimensional sampling function, and where M<<N.The set of sampling functions is then partitioned into K non-overlappingand non-empty subsets of sampling functions {φ₁, . . . , φ_(M1)},{φ_(M1+1), . . . , φ_(M1+M2)}, . . . , {φ_(M−Mk+1), . . . , φ_(M)} eachhaving M_(i) elements, where 1≦i≦K and M₁+M₂+ . . . +M_(k)=M. Each ofthe resulting subset of functions are next arranged in M_(i)×N matricesφ₁, φ₂, . . . , φ_(K). Each matrix has at least Q linearly independentcolumns and at most N−Q linearly dependent columns. An index of thelinearly independent columns is associated with locations correspondingto the ROIs, and an index of the linearly dependent columns isassociated with locations which do not correspond to the ROIs.Thereafter, incoming light is modulated by a spatial light modulatoraccording to a plurality of spatial patterns corresponding to theordered sampling functions. The light reflected/transmitted off themodulator is focused onto a detector of a compressed sensing device. Thedetector proceeds to sample sequential measurements of the light focusedthereon. The measurements comprise a sequence of projection coefficientscorresponding to a scene. Each of the sampled measurements are an innerproduct result y_(m)=<x,φ_(m)>, where x denotes an N-dimensional vectorrepresenting an N-pixel sampled version of a scene. Thereafter, aspatial appearance of the scene is reconstructed from the sequence ofprojection coefficients after M inner products have been sampled. Thereconstructed image being such that pixels associated with the ROIs havea higher image quality when rendered relative to other pixels in theimage.

Features and advantages of the above-described system and method willbecome readily apparent from the following detailed description andaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features and advantages of the subject matterdisclosed herein will be made apparent from the following detaileddescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates one example embodiment of the present method forreconstructing an image of a scene;

FIG. 2 shows one example system for performing scene reconstruction inaccordance with the teachings hereof;

FIG. 3 shows the magnitude of the Haar coefficients of an image x usedin the simulation of the present method in decreasing order, fromlargest to smallest; and

FIG. 4 is a table of results illustrating quality of reconstruction asmeasured by the mean squared error (MSE) from having reconstructed animage with varying number of samples using the present method, as wellas the traditional compressed sensing approach.

DETAILED DESCRIPTION

What is disclosed is a system and method which reconstructs an N-pixelimage of a scene such that Q pixel locations associated with identifiedregions of interest in a scene have a higher image quality when renderedrelative to other pixels in the image.

NON-LIMITING DEFINITIONS

A “region of interest” (ROI) is an identified area of a scene intendedto be reconstructed using the teachings disclosed herein with a higherquality relative to other areas of that scene. What defines a particularregion of interest will largely depend on the application where thepresent invention finds its uses. The image of the scene from which aregion of interest is first identified can be acquired with thecompressed sensing device following the traditional compressed sensingframework. Alternatively, it can be acquired with a traditional imagingdevice having a significantly similar view point of the scene relativeto the compressed sensing device. In yet another embodiment, the imageof the scene can be received from an existing database of images. Aregion of interest is then identified from the image of the scene.Regions of interest can be identified by processing an image of thescene to identify using any of: pixel classification, objectidentification, facial recognition, color, texture, spatial features,spectral information, pattern recognition, motion detection, foregrounddetection, and a user input. The location, shape or size of theidentified region of interest can change over time. Once a region ofinterest has been identified, changes in the location of the identifiedregion of interest can be identified by tracking techniques.Alternatively, re-identification of the region of interest pixelclassification, object identification, facial recognition, color,texture, spatial features, spectral information, pattern recognition,motion detection, foreground detection, and a user input can beperformed periodically.

A “photodetector” or simply “detector” is a device which measures amagnitude of an intensity of light focused thereon. In variousembodiments, the photodetector can be a single (diode) detector or amulti-diode detector and may further comprise an analog-to-digitalconverter and an amplifier.

A “Spatial Light Modulator (SLM)” is a device in the compressed sensingdevice positioned along an optical axis where a camera's focal planearray would typically be located. The SLM is controllable such that itcan be configured according to spatial patterns which can be used tomodulate incoming light which can then be transmitted or reflected ontoa photodetector of a compressed sensing device. As mentioned earlier, acompressed sensing device relies on modulating incoming light from thescene by a spatial light modulator according to a plurality of spatialpatterns. Examples of spatial light modulators include, but are notlimited to Digital Micromirror Devices, Transmissive Liquid Crystals andLiquid Crystals on Silicon.

“Digital Micromirror Device (DMD)” is an optical micro-electromechanical(MEMS) device which has, on its surface, imaging elements comprisingmicroscopic opto-mechanical mirrors arrayed on a two-dimensional grid.Each mirror in the array is referred to as a DMD pixel. The microscopicmirrors are electronically controllable and thus modulate incoming lightby toggling a reflectivity thereof by individually tilting (or rotating)the mirrors in one direction or another to achieve an ON/OFF state. Inthe ON state, light is reflected in a desired direction, such as througha lens or onto a photodetector. In the OFF state, the light is directedelsewhere. By convention, the positive (+) state is ON and the negative(−) state is OFF. The two states are opposite, i.e., if one element is‘1’ then the other is ‘0’, and vice versa. As prescribed by compressedsensing theory, each DMD pattern is configured to select a definedportion of the incoming light onto a detector. During image acquisition,a series of unique patterns are sequentially provided to the DMD and aseries of measurements are obtained. Light energy is reflected by theDMD mirrors onto the photo diode or photoreceptor where the photons ofthe image are converted to an electrical signal. Each signal, producedas a result of each measurement, is a function of a specific pattern andof the scene. By rapidly changing the DMD patterns and obtainingmeasurements therefrom, a time-series signal is obtained. Utilizing acompressed sensing framework, an image reconstruction algorithmreconstructs the original image from the generated time-seriesmeasurement data with knowledge of the temporal sequence of patterns.DMDs are available from vendors in various streams of commerce.

A “Transmissive Liquid Crystal (TLC)” also referred to a “Liquid CrystalModulator (LCM)”, is a programmable array of liquid crystal elements.Each liquid crystal element in the array is a pixel. The liquid crystalelements are individually electronically controllable and thus the TLCmodulates incoming light by toggling a transparency of each TLC pixel toachieve an ON/OFF state. By convention, in the ON state, the liquidcrystal element is transparent so light passes therethrough. In the OFFstate, the liquid crystal element is opaque so light does not passtherethrough. TLCs are desirable in many applications because of theirfast switching times and a high degree of usability over a broad rangeof visible to infrared wavelength bands. TLCs are available from vendorsin various streams of commerce.

A reflective “Liquid Crystal on Silicon (LCOS)” refers to amicro-projection or micro-display technology which uses liquid crystalsinstead of individual mirrors. In LCOS, liquid crystals are applieddirectly to the surface of a silicon chip coated with an aluminizedlayer with some type of passivation layer, which is highly reflective.LCOS technology is preferable in many applications because it canproduce higher resolution and higher contrast images than standardliquid crystal technologies.

A “compressed sensing framework” is a signal processing technique forreconstructing a signal with solutions found by taking advantage of thesignal's sparseness or compressibility in some domain, thereby enablingthe entire signal to be generated from relatively few measurements. Anunderdetermined linear system has more unknowns than equations andgenerally has an infinite number of solutions. In order to choose aproper solution, constraints are applied. Because many signals aresparse, i.e., they contain many coefficients close to or equal to zerowhen represented in some domain, the additional constraint of sparsityallows only those solutions with a small number of non-zero coefficientsto be considered as feasible. Not all underdetermined systems have asparse solution. However, if there is a unique sparse representation tothat underdetermined linear system then a compressed sensing frameworkenables a recovery of that solution.

“Correlation” between two functions φ, Φε

^(N) is defined as the magnitude or absolute value of their innerproduct |

φ,Φ

|, where the inner product

φ,Φ

is obtained by performing N element-wise multiplication and adding the Nindividual results into a single number. Two functions are said to be“largely uncorrelated” if there exists a real number T₁ such that |

φ,Φ

≦T₁. The choice for T₁ is usually application-dependent and is expressedin terms of N and of the value P that bounds Φ and φ, where P is suchthat |Φ(k)|≦P and |φ(k)|≦P, for all k. For example, in one application,two functions are considered to be largely uncorrelated if T₁=0.1NP².Two functions are said to be “largely correlated” if there exists a realnumber T₁′ such that |

φ,Φ

≧T₁′. For example, in one application, two functions are considered tobe largely uncorrelated if T₁′=0.9NP². In the context of the presentdisclosure, a level of correlation between two functions is indicativeof a level of co-linearity: the larger the correlation between twofunctions, the more collinear they are.

“Coherence” between two sets of functions Φ and φ, where Φ={Φ₁, . . . ,Φ_(M1)} with Φ_(i)ε

^(N) for all i, and φ={φ₁, . . . , φ_(M2)} with φ_(j)ε

^(N) for all j is defined as: μ(Φ, φ)=√{square root over(N)}max_(1≦i≦M1, 1≦j≦M2){|

Φ_(i),φ_(j)

}. Two sets of functions Φ and φ are said to be “largely incoherent” ifthere exists a real number T₂ such that μ(Φ, φ)≦T₂. The choice for T₂ isusually application-dependent and is expressed in terms of N and of therange of values of the functions φ_(i) and φ_(j) in Φ and φ, P. Forexample, in one application, two sets are considered to be largelyincoherent for T₂=0.2NP².

Flow Diagram of One Embodiment

Reference is now being made to the flow diagram of FIG. 1 whichillustrates one example embodiment of the present method forreconstructing an image of a scene captured using a compressed sensingdevice with the teachings hereof wherein Q pixel locations associatedwith identified regions of interest in the scene have a higher imagequality when rendered relative to other pixels in the image. Flowprocessing begins at step 100 and immediately proceeds to step 102.

At step 102, receive a mask identifying at least one region of interest(ROI) in a scene. The mask can be dynamically updated in response to anew region of interest having been identified, a location of a region ofinterest changing in a scene, or a user input. The mask may be receivedfrom a remote device over a network via a wired or wireless pathway, orretrieved from a storage device such as a memory or a hard drive. Themask can be a binary image wherein pixels with value ‘1’ (ON pixels)indicate locations associated with the ROI and pixels with value ‘0’(OFF pixels) indicate locations not associated with the ROI. The maskcan also be multi-level, each level corresponding to one identified ROI.

At step 104, order a set of M sampling functions arranged in an M×Nmatrix φ={φ₁, . . . , φ_(M)}, whose m^(th) row vector φ_(m)ε

^(N) denotes the m^(th) N-dimensional sampling function, where M<<N. Theindex m is indicative of an ordering of the function in a samplingsequence. The ordering is such that, given a block length B, partitionsof the set φ into non-overlapping and non-empty subsets definesub-matrices φ₁, φ₂, . . . , φ_(┌M/B┐) all except possibly one of sizeB×N (where ┌┐ denotes the ceiling operator which maps a real number toa smallest following integer) corresponding to sequences of functionsarranged in subsets all except possibly one of length B, {φ₁, . . . ,φ_(B)}, {φ_(B+1), . . . , φ_(2B)}, . . . , {φ_(B┌M/B┐−B+1), . . . ,φ_(M)}, respectively, where B<M is the block length. Each resultingsub-matrix has at least Q linearly independent columns and at most N−Qdependent columns. It should be appreciated that the word “possibly” isused here because either all are of size B×N or all but one are of sizeB×N.

Other partitions of the original set into

$\left\lceil \frac{M}{B} \right\rceil$

subsets, all except one of length B may be utilized. For example, thepartition {φ₁, . . . , φ_(B)}, {φ_(B+1), . . . , φ_(2B)}, . . . ,{φ_(B┌M/B┐−B+1), . . . , φ_(M)} is valid. Partitions of the original setinto

$\left\lfloor \frac{M}{B} \right\rfloor$

subsets, all except possibly one of length B, (where └┘ denotes a flooroperation which maps a real number to a largest previous integer) may beused. It should be appreciated that the word “possibly” is used herebecause either all are of length B or all but one are of length B.

It will be appreciated by someone skilled in the art that otherpartitions into different numbers of subsets each with a possiblydifferent number of elements also serve the purpose of the methodsintroduced in the present disclosure. For example, and more generally(i.e., independently of a block length B), partitions of the set ofsampling functions φ into K non-overlapping and non-empty subsets offunctions φ₁, φ₂, . . . , φ_(K), each subset having M_(i) elements,where 1≦i≦K and M₁+M₂+ . . . +M_(k)=M may also be used. Thecorresponding matrices of size M_(i)×N, where M_(i)<<N, each can have atleast Q linearly independent columns and at most N−Q linearly dependentcolumns.

Since the reconstructed image and the sampling functions have the samenumber of pixels N, there is a one to one correspondence between theindices in the sampling functions and spatial locations in the image.From this correspondence, the index of the linearly independent columnsis determined to be associated with the locations corresponding to theidentified regions of interest, and the index of the linearly dependentcolumns is determined to be associated with the locations notcorresponding to the identified regions of interest.

The value of B determines the difference in the quality ofreconstruction of the ROI vs. non-ROI regions, larger values of Bcorresponding to larger differences. Typical values of B range from 2 to8 although use of values outside that range may be used. In the multipleROI case, each ROI may have an associated B value relative to the commonnon-ROI regions; regions with larger B values will be reconstructed athigher quality relative to regions with smaller B values. For example,if there are two ROIs of sizes Q₁ and Q₂ pixels respectively, and eachassociated with block lengths B₁ and B₂, then let B be the least commonmultiple between B₁ and B₂. Without loss of generality, assume B₂>B₁ andB₂=kB₁ for some integer k so that B=B₂. Each sub-matrix φ₁, φ₂, . . . ,φ_(┌M/B┐) with B rows and N columns will have: a sub-matrix with B rowsand at least Q₂ columns with full rank (e.g., rank B=B₂), an index ofthe linearly independent columns in said sub-matrix being associatedwith locations corresponding to ROI number 2; a sub-matrix with B rowsand at least Q₁ columns with rank B₁=B/k, an index of the columns insaid sub-matrix being associated with locations corresponding to ROInumber 1; and a sub-matrix with B rows and at most N−Q₁−Q₂ columns withrank 1, an index of the linearly dependent columns in said sub-matrixbeing associated with locations which do not correspond to either ROI.In the case where partitions of the set of sampling functions φ into Knon-overlapping and non-empty subsets of functions φ₁, φ₂, . . . ,φ_(K), each subset having M_(i) elements, larger values of M_(i) willlead to larger differences in the quality of reconstruction of ROI vs.non-ROI regions.

Alternatively, the ordering can be such that, for sequences of samplingfunctions arranged in subsets all except possibly one of length B,subsets of sampling functions {φ₁, . . . , φ_(B)}, {φ_(B+1), . . . ,φ_(2B)}, {φ_(B┌M/B┐−B+1), . . . , φ_(M)} are largely incoherent,sampling functions φ_(i) and φ_(j) in a given subset are largelyuncorrelated for every i≠j only for pixels within ROIs as identified bythe mask, and sampling functions φ_(i) and φ_(j) in a given subset arelargely or even completely correlated for pixels outside the ROIs. Inone embodiment, the sampling functions are obtained by adaptivelymodifying a set of largely uncorrelated sampling functions φ={φ₁ . . . ,φ_(M)} (where φ is extracted, for example, from a random matrix withgeneral independent rows or columns, or from a sub-Gaussian matrix,) insuch a way that φ_(i)=φ_(i) for 1≦i≦M at locations associated with theROIs and φ_(i)=φ_(k) for 1≦i≦M and some fixedkε[B┌i/B┐−B+1,B┌i/B┐]∩[1,M], (where ┌┐ denotes the ceiling operatorwhich maps a real number to a smallest following integer) at locationsnot associated with the ROIs. In the general case where partitions ofthe set of sampling functions φ into K non-overlapping and non-emptysubsets of functions φ₁, φ₂, . . . , φ_(K), each subset having M_(i)elements, where 1≦i≦K and M₁+M₂+ . . . +M_(k)=M are used, the samplingfunctions can be obtained by adaptively modifying a set of largelyuncorrelated sampling functions φ={φ₁ . . . , φ_(M)} in such a way thatφ_(i)=φ_(i) for 1≦i≦M at locations associated with the ROIs; forlocations not associated with the ROIs, φ_(i)=φ_(k) for 1≦i≦M, where kis such that when M₁+ . . . +M_(k−1)+1≦i≦M₁+ . . . +M_(k), kε[M₁+ . . .+M_(k−1)+1, M₁+ . . . +M_(k)].

In an alternative embodiment, the sampling functions are obtained byadaptively combining a set of largely uncorrelated sampling functionsφ={φ₁ . . . , φ_(M)} and a set of B-block-wise largely correlatedfunctions α={α₁ . . . , α_(M)}. This is to say that blocks {α₁, . . . ,α_(B)}, {α_(B+1), . . . , α_(2B)}, . . . , {α_(B┌M/B┐−B+1), . . . ,α_(M)} are largely incoherent, and functions α_(i) and α_(j) in a givensubset are largely or fully correlated. The combination is performed insuch a way that φ_(i)=φ_(i) for 1≦i≦M at locations associated with theROIs and φ_(i)=α_(i), for 1≦i≦M and at locations not associated with theROIs. A similar sampling function design strategy can be implemented incases when partitions of the set of sampling functions φ into Knon-overlapping and non-empty subsets of functions φ₁, φ₂, . . . ,φ_(K), each subset having M_(i) elements are used. In this case, α is aset of M_(i)-block-wise largely correlated functions, where blocks {α₁,. . . , α_(M1)}, {α_(M1+1), . . . , α_(M1+M2)}, . . . , {α_(M1−Mk+1), .. . , α_(M)} are largely incoherent, and functions α_(i) and α_(j) in agiven subset are largely or possibly completely correlated. From thesesets of functions, φ_(i)=φ_(i) for 1≦i≦M at locations associated withthe ROIs as before; for locations not associated with the ROIs,φ_(i)=α_(i) for 1≦i≦M.

There are several approaches to obtaining the set of sampling functionsφ with the desired characteristics. In one embodiment, a set of largelyuncorrelated sampling functions φ={φ₁ . . . , φ_(M)} extracted, forexample, from a random matrix with general independent rows or from asub-Gaussian matrix, are adaptively modified such that φ_(i)=φ_(i) for1≦i≦M at locations associated with the ROIs and φ_(i)=φ_(k) for 1≦i≦Mand a fixed kε[B┌i/B┐−B+1,B┌i/B┐]∩[1,M], (where ┌┐ denotes the ceilingoperator which maps a real number to a smallest following integer) atlocations not associated with the ROIs. In cases when partitions of theset of sampling functions φ into K non-overlapping and non-empty subsetsof functions φ₁, φ₂, . . . , φ_(K), each subset having M_(i) elementsare used, φ_(i)=φ_(i) for 1≦i≦M at locations associated with the ROIs;for locations not associated with the ROIs, φ_(i)=φ_(k) for 1≦i≦M, wherek is such that when M₁+ . . . +M_(k−1)+1≦i≦M₁+ . . . +M_(k), kε[M₁+ . .. +M_(k−1)+1, M₁+ . . . +M_(k)].

Measurements are obtained of the scene using a compressed sensing devicewhich comprises, at least in part, a spatial light modulator configuredaccording to a plurality of spatial patterns corresponding to the set ofsampling functions. Each pattern focuses a portion of incoming lightonto a detector which samples sequential measurements of light focusedthereon. Each of the measurements is an inner product resulty_(m)=<x,φ_(m)>, where x denotes an N-dimensional vector representingthe N-pixel sampled version of the scene. A series of measurementscomprises a sequence of projection coefficients corresponding to theinner product between that N-pixel sampled version of the scene and eachof the sampling functions. An appearance of the scene is reconstructedfrom the sequence of projection coefficients after M inner products havebeen sampled, where M<<N, such that pixel locations associated with ROIsin the scene have a higher image quality when rendered relative to otherpixels in the image.

The reason that this method is advantageous, particularly in the contextof a DMD-based single pixel camera, has to do with the way a DMD moduleswitching occurs. The switching capabilities of the array are limited sothat instead of switching all individual micromirrors simultaneously, agroup of micromirrors (e.g., a column, row or more generally, a cluster)has to be switched before the next group can be switched. Sincesingle-pixel camera measurements cannot be taken until all micromirrorsin the array have switched to their intended position, this sequentialswitching introduces undesirable delays and, consequently, reduces theeffective sampling rate of the camera. In the present approach, sinceonly a fraction of the micromirrors is switched at every cycle, theeffect of the sequential switching mode on the sampling rate isameliorated by a factor approximately equal to the ratio of the numberof non-ROI pixels to the number of ROI pixels. Note that performingmeasurements with the ordered sampling functions obtained in the mannerdescribed above, achieves reconstruction of an ROI with increasedquality relative to the rest of the image; additionally, and in thecontext of a DMD-based single pixel camera it also enables fastersampling rates due to the way a DMD module switching occurs, as statedabove.

At step 106, configure a spatial light modulator to modulate incominglight according to spatial patterns corresponding to the set of orderedsampling functions.

At step 108, use a detector to sequentially sample the focused spatialpattern of light to obtain measurements of the scene, each comprising aninner product result y_(m)=<x,φ_(m)>, where x denotes an N-dimensionalvector representing the N-pixel sampled version of the scene (i.e., thevectorized matrix representation of an image). The measurements comprisea sequence of projection coefficients corresponding to the scene.

At step 110, reconstruct a spatial appearance of an image of the scenefrom the sequence of projection coefficients after M inner products havebeen sampled. The reconstruction is such that pixels associated with thelocalized ROIs in the reconstructed image have a higher image quality,that more faithfully represents the appearance of the sampled scene,when rendered relative to other pixels in the image. This non-uniformquality of the reconstruction is enabled by the sampling schemedescribed above. Thereafter, in this embodiment, further processingstops.

It should be appreciated that the flow diagrams hereof are illustrative.One or more of the operations illustrated in the flow diagrams may beperformed in a differing order. Other operations may be added, modified,enhanced, or consolidated. Variations thereof are intended to fallwithin the scope of the appended claims.

Example System Architecture

Reference is now being made to FIG. 2, which shows one example systemfor performing scene reconstruction in accordance with the teachingshereof.

In the system of FIG. 2, incoming light (collectively at 201) enters thecompressed sensing system 200 through an aperture 202 and into a spatiallight modulator (SLM) 203 which modulates the incoming light to producea spatial pattern of light 204 which is focused on to detector 205. Thedetector measures a magnitude of an intensity of the spatial patternfocused thereon. Mask module 208 receives a mask using USB port 209, andprovides the mask to a controller 207 shown comprising a processor (CPU)and a memory. The controller facilitates the configuration of thespatial light modulator to modulate incoming light. The detector outputssequential measurements 206 which may be provided to USB port 215.Measurements obtained by the detector are communicated to imagereconstruction module 213 wherein a spatial appearance of the scene isreconstructed from the measurements. The measurements and thereconstructed image 214 are communicated to storage device 216 and/orprovided as output to workstation 220. Values, data, measurements, andresults of any of the modules and processing units of the system 200 maybe obtained or retrieved via communications bus 217.

Shown in communication with the system 200 is a workstation 220. Theworkstation is shown comprising a monitor 221, a keyboard 222, a mouse223, a storage device 224, and a computer-readable media 225. Theworkstation is also placed in communication with one or more remotedevices over network 226 using, for example, a network card. A userthereof may change or control the functionality of any of the modulesand processing units comprising the system 200 using the workstation. Animage of a scene can be displayed on the monitor and corrected and/orcropped. Masks can be generated using the workstation and communicatedto the mask module 208. Measurements and values generated by the systemmay be displayed on the display device. Intensity values obtained by thedetector may be modified by a user of the workstation. The values of thespatial pattern that controls the modulation of incoming light mayfurther be communicated to the workstation and displayed on the monitor.A user can selectively identify regions of interest using, for example,a mouse. The user may further define the ordering of the set of samplingfunctions using the workstation. Localized regions of interest can becommunicated to the mask module by the workstation. An operator of theworkstation may modify the results generated by any of the modules orprocessing units of FIG. 2 as needed and/or re-direct the modifiedresults back to the same or different modules for further processing orre-processing. It should be appreciated that the workstation has anoperating system and other specialized software configured to display avariety of numeric values, text, scroll bars, pull-down menus with userselectable options, and the like, for entering, selecting, or modifyinginformation displayed on the display device.

Various modules may designate one or more components which may, in turn,comprise software and/or hardware designed to perform the intendedfunction. A plurality of modules may collectively perform a singlefunction. Each module may have a specialized processor capable ofexecuting machine-readable program instructions. A module may comprise asingle piece of hardware such as an ASIC, electronic circuit, or specialpurpose processor. A plurality of modules may be executed by either asingle special purpose computer system or a plurality of special purposecomputer systems in parallel. Modules may include software/hardwarewhich may further comprise an operating system, drivers, controllers,and other apparatuses some or all of which may be connected via anetwork.

Implementation Details

Compressed sensing deals with signal recovery from highly incompleteinformation. A cornerstone of compressed sensing is that anN-dimensional sparse vector x[] can be recovered from a small number Mwhere M<<N of linear measurements y_(m)=<x,φ_(m)>, m=1, 2, . . . , M,under a certain set of assumptions and by solving a convex optimizationproblem. In matrix form, y=φx where x=Ψs, with s being a sparse vectorand Ψ defining a transformation to a domain where x is sparse; Ψ can be,for example, an orthonormal transform such as a DCT, wavelet or FFTtransform. The number of non-zero entries in s determines the degree ofsparseness K of x. For testing purposes, a set of binary pseudo-randomsampling functions were used as sampling matrix φ of an image x, alongwith the assumption that each measurement (i.e., the inner productresult) was uniformly quantized to a certain number of bits, and thatthe image was sparse in the Haar wavelet domain; note that assumptionsof the image being sparse in other domains are anticipated. FIG. 3 showsthe magnitude of the Haar coefficients of an image x used in thesimulation of the present method in decreasing order, from largest tosmallest. It can be seen that about 80% of the image energy is containedin the largest 500 Haar coefficients, while 99.9% of the image energy iscontained in the largest 3000 coefficients.

The test image was sampled according to the teachings herein, and eachof the measurements or inner products, quantized or digitized to a fixedbit depth. From the quantized set of measurements, we posed the recoveryproblem as an optimization x=argmin{∥x∥₁} subject to y=φx where, ∥∥₁denotes the l₁ norm. It should be appreciated that this is only one ofmultiple ways to solve an inverse problem. Other approaches includeexploiting the assumption that the gradient of the image is sparse, aswell as building over-complete dictionaries in which the representationtarget image is sparse. The method described herein is equally adaptableto these techniques as well, as they rely on the use of sequences ofsampling functions to perform the sensing.

An embodiment of the present system was used to reconstruct a spatialappearance of a scene comprising a 64×64 pixel image of the scene. Abinary mask was used to identify an example ROI with dimensions 32×64pixels. The table of FIG. 4 shows the overall mean squared error as wellas the mean squared error between an image reconstructed from variousnumbers of samples and various block lengths B using the present method.That is, the set of sampling function was partitioned into subsets ofthe form {φ₁, . . . , φ_(B)}, {φ_(B+1), . . . , φ_(2B)}, . . . ,{φ_(B┌M/B┐−B+1), . . . , φ_(M)}.

It can be seen by an examination of the results thereof that, as theblock length increases, the quality of the reconstructed image asmeasured by the mean square error decreases; however, the quality of thereconstructed ROI suffers little for the range of B values tested, and,it is consistently better or on par relative to that yielded by thetraditional compressed sensing approach. These results showcase theefficiency of the present method in preserving the quality of thetargeted region of interest as the lengths of the blocks increase; werefer to this as improved encoding efficiency, since the quality of thereconstructed ROI is better than that yielded by traditional compressedsensing methods for a given number of measurements. Note that, in thecontext of a single pixel camera, use of the present method would resultin increased switching frequencies: the larger the value of B, thefaster the subsequent samples can be acquired.

One or more aspects of the teachings disclosed herein are intended to beincorporated in an article of manufacture. The above-disclosed featuresand functions or alternatives thereof, may be combined into othersystems or applications. Presently unforeseen or unanticipatedalternatives, modifications, variations, or improvements may becomeapparent and/or subsequently made by those skilled in the art and,further, may be desirably combined into other different systems orapplications. Changes to the above-described embodiments may be madewithout departing from the spirit and scope of the invention. Theteachings of any printed publications including patents and patentapplications, are each separately hereby incorporated by reference intheir entirety.

What is claimed is:
 1. A method for reconstructing an N-pixel image of ascene captured using a compressed sensing device, the method comprising:receiving a mask identifying at least one region of interest (ROI) ofsize Q pixels in the scene; arranging a set of M sampling functions inan M×N matrix φ={φ₁, . . . , φ_(M)}, whose m^(th) row vector φ_(m)ε

^(N) denotes the m^(th) N-dimensional sampling function, and where M<<N;partitioning said set of sampling functions into K non-overlapping andnon-empty subsets of functions, each subset having M_(i) elements, where1≦i≦K and M₁+M₂+ . . . +M_(k)=M; arranging each resulting subset offunctions in M_(i)×N matrices φ₁, φ₂, . . . , φ_(K), each matrix havingat least Q linearly independent columns and at most N−Q linearlydependent columns, an index of said linearly independent columns beingassociated with locations corresponding to said ROIs, and an index ofsaid linearly dependent columns being associated with locations which donot correspond to said ROIs; modulating, by a spatial light modulator,incoming light according to a plurality of spatial patternscorresponding to said ordered sampling functions, said modulated lightbeing focused onto a detector of a compressed sensing device; sampling,by said detector, sequential measurements of said modulated lightfocused thereon, each of said sampled measurements being an innerproduct result y_(m)=<x,φ_(m)>, where x denotes an N-dimension vectorrepresenting the N-pixel sampled version of said scene, saidmeasurements comprising a sequence of projection coefficientscorresponding to said scene; and reconstructing a spatial appearance ofsaid scene from said sequence of projection coefficients after M innerproducts have been sampled, said reconstructed image being such thatpixels associated with said ROIs have a higher image quality whenrendered relative to other pixels in said image.
 2. The method of claim1, wherein said partition is performed according to a block size Bdetermining one of two possible partitions, φ₁, φ₂, . . . , φ_(┌M/B┐)and φ₁, φ₂, . . . , φ_(┌M/B┐) so that K equals one of ┌M/B┐ and ┌M/B┐,wherein each subset in the partition except possibly one has B elements.3. The method of claim 2, wherein said partition according to a blocksize B is one of: {φ₁, . . . , φ_(B)}, {φ_(B+1), . . . , φ_(2B)}, . . ., {φ_(B┌M/B┐−B+1), . . . , φ_(M)}, and {φ₁, . . . , φ_(B)}, {φ_(B+1), .. . , φ_(2B)}, . . . , {φ_(B┌M/B┐−2B+1), . . . , φ_(M)}, where M>2B. 4.The method of claim 1, wherein subsets of sampling functions φ₁, φ₂, . .. , φ_(K) are largely incoherent, sampling functions φ_(i) and φ_(j) ina given subset are largely uncorrelated for every i≠j only for indicesassociated with said ROIs, and sampling functions φ_(i) and φ_(j) in agiven subset are largely correlated for indices not associated with saidROIs.
 5. The method of claim 4, wherein said sampling functions areobtained by adaptively modifying a set of largely uncorrelated samplingfunctions φ={φ₁, . . . , φ_(M)} in such a way that φ_(i)=φ_(i) for 1≦i≦Mat locations associated with said ROIs, and φ_(i)=φ_(k) for 1≦i≦M wherek is such that when M₁+ . . . +M_(k−1)+1≦i≦M₁+ . . . +M_(k), kε[M₁+ . .. +M_(k−1)+1, M₁+ . . . +M_(k)] at locations not associated with saidROIs.
 6. The method of claim 2, wherein said sampling functions areobtained by adaptively modifying a set of largely uncorrelated samplingfunctions φ={φ₁ . . . , φ_(M)} in such a way that φ_(i)=φ_(i) for 1≦i≦Mat locations associated with said ROIs and φ_(i)=φ_(k) for 1≦i≦M andkε[B┌i/B┐−B+1,B┌i/B┐]∩[1,M], where ┌┐ denotes the ceiling operatorwhich maps a real number to a smallest following integer at locationsnot associated with said ROIs.
 7. The method of claim 2, wherein saidsampling functions are obtained by adaptively combining a set of largelyuncorrelated sampling functions φ={φ₁ . . . , φ_(M)} and a set ofB-block-wise largely correlated functions α={α₁ . . . , α_(M)} in such away that φ_(i)=φ_(i) for 1≦i≦M at indices associated with said ROIs, andφ_(i)=α_(i) for 1≦i≦M at indices not associated with said ROIs.
 8. Themethod of claim 1, wherein said sampling functions are generated basedon an output of a random number generator, wherein generated values ofsampling functions associated with said ROIs update at a different ratethan those not associated with said ROIs, said output of said randomnumber generator being passed through a deterministic function in orderto obtain samples from different distributions.
 9. The method of claim1, wherein said spatial light modulator comprises any of: a digitalmicromirror device, a transmissive liquid crystal, and reflective liquidcrystal on silicon.
 10. The method of claim 1, further comprisingprocessing an image of said scene to identify said ROIs using any of:pixel classification, object identification, facial recognition, color,texture, spatial features, spectral information, pattern recognition,motion detection, foreground detection, and a user input
 11. The methodof claim 1, further comprising processing an image of said scene togenerate said mask wherein pixels associated with said ROI as beingactive and pixels outside said ROI as being inactive.
 12. The method ofclaim 1, wherein said mask is updated in real-time in response to alocation of any of said ROIs changing over time.
 13. The method of claim2, wherein different ROIs are associated with different subset lengthsB.
 14. A compressed sensing system for reconstructing an N-pixel imageof a scene captured using a compressed sensing device, the systemcomprising: a mask identifying at least one Q-pixel region of interest(ROI) in a scene; a spatial light modulator for modulating incominglight according to spatial patterns corresponding to a set of M samplingfunctions arranged in an M×N matrix φ={φ₁, . . . , φ_(M)}, whose m^(th)row vector φ_(m)ε

^(N) denotes the m^(th) N-dimensional sampling function, and where M<<N,said set of sampling functions being partitioned into K non-overlappingand non-empty subsets of functions, each subset having M_(i) elements,where 1≦i≦K and M₁+M₂+ . . . +M_(k)=M, each resulting subset offunctions being arranged in M_(i)×N matrices φ₁, φ₂, . . . , φ_(K), eachmatrix having at least Q linearly independent columns and at most N−Qlinearly dependent columns, an index of said linearly independentcolumns being associated with locations corresponding to said ROIs, andan index of said linearly dependent columns being associated withlocations which do not correspond to said ROIs; a detector of acompressed sensing device, said detector sampling sequentialmeasurements of said modulated light focused thereon, each of saidsampled measurements being an inner product result y_(m)=<x,φ_(m)>,where x denotes an N-dimension vector representing the N-pixel sampledversion of a scene, said measurements comprising a sequence ofprojection coefficients corresponding to said scene; and a processorexecuting machine readable program instructions for reconstructing aspatial appearance of said scene from said sequence of projectioncoefficients after M inner products have been sampled, saidreconstructed image being such that pixels associated with said ROIshave a higher image quality when rendered relative to other pixels insaid image.
 15. The system of claim 14, wherein said partition isperformed according to a block size B determining one of two possiblepartitions, φ₁, φ₂, . . . , φ_(┌M/B┐) and φ₁, φ₂, . . . , φ_(┌M/B┐) sothat K equals one of ┌M/B┐ and ┌M/B┐, wherein each subset in thepartition except possibly one has B elements.
 16. The system of claim 15wherein said partition according to a block size B is one of: {φ₁, . . ., φ_(B)}, {φ_(B+1), . . . , φ_(2B)}, . . . , {φ_(B┌M/B┐−B+1), . . . ,φ_(M)}, and {φ₁, . . . , φ_(B)}, {φ_(B+1), . . . , φ_(2B)}, . . . ,{φ_(B┌M/B┐−2B+1), . . . , φ_(M)}, where M>2B.
 17. The system of claim14, wherein subsets of sampling functions φ₁, φ₂, . . . , φ_(K) arelargely incoherent, sampling functions φ_(i) and φ_(j) in a given subsetare largely uncorrelated for every i≠j only for indices associated withsaid ROIs, and sampling functions φ_(i) and φ_(j) in a given subset arelargely correlated for indices not associated with said ROIs.
 18. Thesystem of claim 17, wherein said sampling functions are obtained byadaptively modifying a set of largely uncorrelated sampling functionsφ={φ₁ . . . , φ_(M)} in such a way that φ_(i)=φ_(i) for 1≦i≦M atlocations associated with said ROIs, and φ_(i)=φ_(k) for 1≦i≦M where kis such that when M₁+ . . . +M_(k−1)+1≦i≦M₁+ . . . +M_(k), kε[M₁+ . . .+M_(k−1)+1, M₁+ . . . +M_(k)] at locations not associated with saidROIs.
 19. The system of claim 15, wherein said sampling functions areobtained by adaptively modifying a set of largely uncorrelated samplingfunctions φ={φ₁ . . . , φ_(M)} in such a way that φ_(i)=φ_(i) for 1≦i≦Mat locations associated with said ROIs and φ_(i)=φ_(k) for 1≦i≦M andkε[B┌i/B┐−B+1,B┌i/B┐]∩[1,M], where ┌┐ denotes the ceiling operatorwhich maps a real number to a smallest following integer at locationsnot associated with said ROIs.
 20. The system of claim 15, wherein saidsampling functions are obtained by adaptively combining a set of largelyuncorrelated sampling functions φ={φ₁ . . . , φ_(M)} and a set ofB-block-wise largely correlated functions α={α₁ . . . , α_(M)} in such away that φ_(i)=φ_(i) for 1≦i≦M at indices associated with said ROIs, andφ_(i)=α_(i) for 1≦i≦M at indices not associated with said ROIs.
 21. Thesystem of claim 14, wherein said sampling functions are generated basedon an output of a random number generator, wherein generated values ofsampling functions associated with said ROIs update at a different ratethan those not associated with said ROIs, said output of said randomnumber generator being passed through a deterministic function in orderto obtain samples from different distributions.
 22. The system of claim14, wherein said spatial light modulator comprises any of: a digitalmicromirror device, a transmissive liquid crystal, and reflective liquidcrystal on silicon.
 23. The system of claim 14, further comprisingprocessing an image of said scene to identify said ROIs using any of:pixel classification, object identification, facial recognition, color,texture, spatial features, spectral information, pattern recognition,motion detection, foreground detection, and a user input
 24. The systemof claim 14, further comprising processing an image of said scene togenerate said mask wherein pixels associated with said ROI as beingactive and pixels outside said ROI as being inactive.
 25. The system ofclaim 14, wherein different ROIs are associated with different subsetlengths B.